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Aims Through a 4-year follow-up of the abstracts submitted to the European Society of Cardiology Congress in 2006, we 

aimed at identifying factors predicting high-quality research, appraising the quality of the peer review and editorial 
processes, and thereby revealing potential ways to improve future research, peer review, and editorial work. 

Methods All abstracts submitted in 2006 were assessed for acceptance, presentation format, and average reviewer rating. 

and results Accepted and rejected studies were followed for 4 years. Multivariate regression analyses of a representative selec- 

tion of 10% of all abstracts (n = 1002) were performed to identify factors predicting acceptance, subsequent publi- 
cation, and citation. A total of 10 020 abstracts were submitted, 3104 (31%) were accepted for poster, and 701 (7%) 
for oral presentation. At Congress level, basic research, a patient number > 100, and prospective study design were 
identified as independent predictors of acceptance. These factors differed from those predicting full-text publication, 
which included academic affiliation. The single parameter predicting frequent citation was study design with rando- 
mized controlled trials reaching the highest citation rates. The publication rate of accepted studies was 38%, whereas 
only 24% of rejected studies were published. Among published studies, those accepted at the Congress received 
higher citation rates than rejected ones. 

Conclusions Research of high quality was determined by study design and largely identified at Congress level through blinded peer 

review. The scientometric follow-up revealed a marked disparity between predictors of full-text publication and 
those predicting citation or acceptance at the Congress. 
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Introduction 

The Congress of the European Society of Cardiology (ESC) is one 
of the largest international scientific meetings worldwide, yearly 
attracting approximately 30 000 health care professionals from 
more than 60 countries. 1,2 Scientific meetings like this provide an 
excellent forum for distribution and discussion of the most 
recent scientific findings. However, the factors that determine ac- 
ceptance at a scientific meeting, and whether these factors also 
predict subsequent full-text publication are largely unknown. More- 
over, a validation of the parameters that predict acceptance at a sci- 
entific meeting or acceptance for full-text publication with respect 
to their later impact has not been performed. 

Several studies have investigated distinct aspects such as positive 
outcome- or institutional bias associated with the acceptance at 
scientific meetings or publication in peer review journals. 3-6 Few 
studies have assessed the publication fate of abstracts submitted 
to scientific meetings. 7-9 In the field of cardiology, Toma et a!. 10 
specifically investigated the fate of randomized controlled trials 
(RCTs) presented at the scientific meetings of the American 
College of Cardiology between 1999 and 2002. Ross et al. evalu- 
ated the effects of blinded peer review for abstracts submitted 
to the Scientific Sessions of the American Heart Association 
from 2000 to 2004. 10 ' 11 To date, no study has followed the 
same cohort of abstracts from their submission to a scientific 
meeting over their full-text publication up to their subsequent 
impact after publication. Without an external and neutral surrogate 
for scientific quality such as citation analysis, earlier studies could 
not appraise whether the observed effects affected later scientific 
impact. 

Our objective was to identify distinct factors predicting accept- 
ance at an international scientific meeting in the field of cardiology 
as well as parameters predicting subsequent full-text publication 
and citation. Beyond prediction at each of these levels, we aimed 
to validate the parameters predicting acceptance at Congress 
level or publication with those parameters predicting subsequent 
scientific impact, thereby appraising the quality of the peer 
review and editorial processes involved at each stage. 

To accomplish these goals, we assessed all 10 020 abstracts sub- 
mitted to the Congress of the ESC in 2006 (i.e. joint meeting of the 
ESC and the World Heart Foundation) and followed a representa- 
tive sample of 10% (n = 1002 abstracts) of both accepted and 
rejected abstracts over a period of 4 years for full-text publication 
and subsequent 2-year citations as a surrogate for scientific impact. 

Methods 

Study design and categorization of abstracts 

All 10 020 abstracts submitted to the ESC congress in 2006 were 
entered into a database and assessed for acceptance, presentation 
format (oral vs. poster presentation), and the average reviewer 
rating. For a precise assessment and scientometric follow-up, a 
computer-assisted random selection of 10% (n= 1002, margin of 
error = 0.01) 12 of all abstracts submitted was assessed in detail 
according to a set of pre-specified variables. These variables were 
origin and type (clinical vs. basic) of research, origin and type 
(university-affiliated vs. not university-affiliated) of institution, study 



design (RCTs vs. prospective non-randomized trials vs. retrospective 
studies vs. systematic reviews vs. meta-analyses), the number of 
patients enrolled in clinical studies, the study field of clinical research, 
and the gender of first and last authors. The reliability of the computer- 
assisted representative random selection was confirmed by comparing 
the distribution of variables common to both the original sample of all 
10 020 abstracts and the randomly selected sample of 1002 abstracts; 
these common variables were acceptance at the Congress (accepted 
vs. rejected), presentation format, and average reviewer rating. No dif- 
ferences were observed (see Supplementary material online, Table S4). 
Gender of first and last authors was assessed using a standardized 
algorithm based on the analysis of their first and surnames in conjunc- 
tion with their country of origin. In specific cases, this approach was 
supplemented using online searches for photographs associated with 
their home institution when necessary and available. This algorithm 
is described in detail in the online supplementary material and was 
developed a priori. Given the need to use this empirical approach, 
however, the analyses that included gender should be considered 
exploratory. 

Scientometric follow-up for publication 
and citation 

The representative random selection of all abstracts submitted (includ- 
ing both accepted and rejected abstracts) was followed for the time 
point and journal of publication and for the number of subsequent 
2-year citations, beginning from the time of publication. Abstracts 
were followed for 4 years. The impact factor of the journal of publica- 
tion was assessed for the year of publication. To identify whether an 
abstract had been published, the following algorithm was used: first, 
the full abstract title was entered as search term in PubMed®. If no 
publication was identified, a shortened version of the abstract title 
(keywords only, no adjectives) was entered. If still no publication 
was identified, the full list of authors was entered in PubMed®, 
which, in case of no match, was followed by the names of only the 
first and last authors. If again no publication was identified, the same 
algorithm was repeated first in Google Scholar, then in Google, each 
indexing increasing amounts of contents. Thus, articles published in 
journals not indexed in Pubmed® were identified. The Cochrane 
library was searched for abstracts of systematic reviews. To validate 
that identified articles and the respective abstracts matched, the fol- 
lowing parameters were compared for every full-text article that 
was identified: primary endpoint(s), the number and size of the 
study groups, and the conclusions. Articles assessing the same end- 
points in the same number of study groups of the same or slightly 
increased sizes, leading to the same conclusions, and published by 
the same last authors compared with the respective abstract were 
considered 'published'. The number of 2-year citations of original arti- 
cles (beginning from the time of publication) was determined using the 
citation indexing and search service of the ISI (Institute of Scientific In- 
formation) Web of Science (Thomson Reuters). 13 Data were collected 
by four independent reviewers (S.W., D.A.R., J.H.W., M.H.) in January 
and February 2011, and updated in December 2011 for completion of 
the follow-up of 2-year citations. In a pilot study, 3% of the study 
samples (n = 30) were assessed independently by each of the four 
reviewers to address inter observer variances. Thereafter, one out 
of the four reviewers assessed each of the 1002 abstracts. The 
cut-off for 'high' 2-year citation rates was set to 10 or more (>85th 
percentile of all published studies) in order to identify the top 15% 
of published articles. Receiver operating characteristic (ROC) curve 
analyses were performed to evaluate the predictive value of average 
reviewer ratings for frequent 2-year citation. 
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Regression analyses 

Univariable and multivariable binary logistic regression analyses were 
performed to identify factors predicting acceptance at Congress 
level, as well as subsequent publication and citation frequency. Multi- 
variable regression analyses were run using backward conditional vari- 
able exclusion and list-wise case exclusion. The threshold for entry 
was 0.05 and for exclusion 0.10. 

Statistical analyses 

Metric variables were assessed for distribution using the Kolmogorov- 
Smirnov test and compared by the Mann-Whitney U, one-way 
ANOVA, and Kruskal-Wallis H-tests where appropriate. Significance 
levels were adjusted for multiple comparisons using conservative Bon- 
ferroni's correction where indicated. Proportional differences derived 
from categorical data were compared using the Fisher exact or x 2 tests 
where appropriate and presented as odds ratio and 95% confidence 
interval. P-values were two-sided and considered statistically significant 
for P < 0.05, if not indicated differently (multiple comparisons). Statis- 
tical analyses were performed with IBM SPSS Statistics Version 19. 

Results 

From abstract at the ESC congress 
to publication 

The 10 020 abstracts submitted to the ESC congress in 2006 from 
63 different countries of all five continents gave rise to over 2900 
publications and over 32 000 citations (Figures 1 and 2, Supplemen- 
tary material online, Table S2). Of all abstracts submitted 90% were 
clinical studies and 10% were basic research. Nearly one-fifth of 
the research submitted had been performed in institutions not 
affiliated to universities (Table 1). Half of the clinical studies submit- 
ted were retrospective, 35% were designed prospectively, of which 
18% were RCTs (Table 1). In a pilot study encompassing 3% of the 
study sample (n = 30), inter-observer agreement was assessed. 
Fleiss' kappa values for categorical variables ranged from 0.853 
to 1.0, and intraclass correlation coefficients for continuous vari- 
ables from 0.996 to 1.0 (see Supplementary material online, 
Table S5). 

Peer review and editorial process at the 
ESC Congress 2006 

All abstracts submitted were reviewed by three to eight independ- 
ent reviewers, who allotted scores from 1 to 10 for each abstract. 
Editorial decisions were based on average reviewer ratings. Re- 
viewer ratings showed a central tendency without being normally 
distributed (see Supplementary material online, figure SJ). 
Abstracts accepted had received significantly higher scores than 
those rejected (P < 0.001). Among accepted studies, those 
accepted for oral presentation had been rated significantly better 
than those accepted for poster presentation (P < 0.001), indicating 
a good coherency of reviewer assessment and editorial decisions. 
A cut-off for acceptance was identified at an average reviewer 
rating of 6.5 (Figure 3A). The predictive value of expert reviewer 
assessment for frequent citation (defined as 10 or more 2-year 
citations) was 70% (AUC = 0.694, figure 36). The best cut-off 
for frequent citation was identified to be an average reviewer 
rating of greater than 7 (AUC = 0.660, P< 0.001, Table 2). 



Concomitantly, studies accepted at the ESC Congress received sig- 
nificantly more 2-year citations than rejected ones (P = 0.004 for 
oral presentations, P = 0.008 for poster presentations, figure 3D). 

Factors predicting acceptance at the 
ESC Congress 

Thirty-eight per cent of all submitted studies were accepted, and 
only 7% of all submitted abstracts (i.e. 18.4% of all accepted 
abstracts) were accepted for oral presentation (Figure J). 

In univariable analyses, two factors were positively associated 
with acceptance at Congress level: study design (P = 0.018) and a 
patient number of 100 or more (OR = 1.93, P< 0.001). Within 
study design, specifically prospective non-randomized study design 
(OR =1.90, P = 0.016) and randomized controlled trial design 
(OR 1.56, P = 0.004) were identified. The only factor not favouring 
acceptance in univariable analysis was research conducted in the 
field of rhythmology (OR = 0.57, P = 0.034) with an acceptance 
rate of 33% compared with 46% of other abstracts (Table 1). 

When adjusting for all factors assessed, the number of patients 
enrolled showed the strongest association with acceptance 
(adjusted OR = 2.07, P< 0.001), followed by basic as opposed 
to clinical research (adjusted OR = 2.24, P = 0.001) with an ac- 
ceptance rate of 44 vs. 37%, and study design (P = 0.023). 
Within study design, prospective data collections (i.e. prospective 
cohort studies and non-randomized intervention trials) (adjusted 
OR =1.67, P = 0.002) and RCTs (adjusted OR =1.91, P = 
0.020) favoured acceptance (Table J). Interestingly, none of the 
aspects assessed was associated with oral or poster presentation 
among all accepted abstracts (see Supplementary material online, 
Table S1). 

Factors predicting full-text publication 

Of all accepted studies, 38% were subsequently published, whereas 
only 24% of rejected studies were published (P < 0.001). The pub- 
lication rate of studies accepted for oral presentation (37%) did not 
differ from the publication rate of studies accepted for poster pres- 
entation (39%, P = 0.79). However, significantly more of those 
studies that had been accepted for oral presentation were subse- 
quently published in high impact factor journals (30%, IF > 10) 
compared with studies that had been accepted for poster presen- 
tation (10%, P = 0.009) or rejected studies (4%, P< 0.001) 
(figure ?). 

Univariable analyses revealed a significant advantage for institu- 
tions affiliated with universities to publish their research (publica- 
tion rate 31%) compared with non-university-affiliated 
institutions (publication rate = 22%, Table 3). With a publication 
rate of 41%, basic research showed the strongest association 
with publication (OR =1.72, P = 0.012), followed by study 
design in clinical research (P = 0.014) and the affiliation to a univer- 
sity (OR = 1.53, P = 0.025). Notably, neither prospective nor ran- 
domized controlled study design, but meta-analyses contributed to 
the significant association of study design with publication in uni- 
variable analysis (OR = 5.97, P = 0.041). However, the small 
number of meta-analyses in the studied cohort does not allow a 
solid statement. 
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Figure I Fate of studies submitted to the ESC Congress 2006. Overview of submission, acceptance at Congress level, and follow-up for 
full-text publication, and citation within 2 years following publication. All abstracts were characterized according to a set of pre-specified para- 
meters. Association of these factors with the acceptance at Congress level, with full-text publication, and subsequent citation was assessed by 
univariable and mutlivariable regression analyses. Percentages refer to the respective preceding level (n). (n* refers to the representative 
random selection of 10% (1002) of all abstracts submitted (10 020), 0 published, not published; IF, impact factor). 



When adjusting for all factors assessed, prospective non- 
randomized study design independently predicted publication 
(adjusted OR = 1.44, P = 0.032). Moreover, multivariable analyses 
confirmed university affiliation to independently predict publica- 
tion (adjusted OR = 1.62, P = 0.018). The number of patients en- 
rolled was not associated with full-text publication, while it was 
with acceptance at Congress level. 

Factors predicting citation 

Twenty-one per cent of all studies that had been accepted and sub- 
sequently published were cited 10 times or more within the 
2 years following publication. In contrast, only 7% of the published 
studies that had been rejected were cited 10 times or more in the 
2 years following publication (P = 0.008). Citation rates between 
published studies that had been accepted for oral presentation 
did not differ significantly from those that had been accepted for 
poster presentation (P = 0.24). In univariable analyses, the study 
field (P = 0.028) and study design (P = 0.001) were associated 
with high citation rates (Table 4). After adjusting for the remaining 
factors, study design remained the only parameter independently 
associated with high citation rates (P = 0.004). In particular, 



randomized controlled trial design demonstrated a marked inde- 
pendent association with high citation rates (adjusted OR = 6.82, 
P< 0.001), followed by prospective non-randomized studies 
(adjusted OR = 2.57, P = 0.018). Neither academic affiliation nor 
gender of first or last author, nor the number of enrolled patients 
influenced citation rates {Table 4). 

At Congress level, editorial decisions correlated both with iden- 
tified predictors of high citation rates, i.e. study design, and with re- 
viewer ratings (Figure 3A). Notably, odds ratios for randomized 
controlled trial design differed by over three-fold between the ac- 
ceptance at Congress level and high citation rates. Along this line, 
there was no significant difference in the number of 2-year cita- 
tions between studies accepted for oral and those accepted for 
poster presentation (Figure 3D). Over half of the submitted 
RCTs were rejected at the ESC congress. However, RCTs still 
reached the highest acceptance rate of all factors assessed 
(Table 1). Over 80% of the accepted RCTs were accepted for 
poster presentation (see Supplementary material online, Table 
S1). It is consequently not surprising that a few of the most 
highly cited studies evolved from abstracts originally accepted for 
poster presentation (Figure 3D). 
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Figure 2 Origin of research submitted to the ESC Congress 2006. A total of 10 020 abstracts were submitted to the scientific sessions of the 
ESC 2006. Based on the assessment of a representative subsample of 10% (n = 1002) submitted studies originated from 63 different countries 
of all five continents. About one-third of all studies were submitted from the American continent, 43% from Western Europe, and 14% from 
Eastern Europe and Russia. Less than 10% of studies originated from Africa and Asia/Oceania. 



Does female gender affect scientific 
success? 

As noted earlier, these analyses should be considered exploratory 
given that information on gender was not directly available but had 
to be empirically determined through an examination of first and 
last author names. Of all abstracts submitted, 26% were conducted 
by female first authors (Table 1). Only 13% of senior authors were 
female, suggesting that both female junior and senior scientists are 
markedly underrepresented in the cardiovascular research commu- 
nity. Acceptance rates at Congress level did not differ between 
female and male scientists (Table 1, Supplementary material 
online, Table S3). Among junior researchers, publication rates did 
not differ markedly between male and female colleagues (30 vs. 
27%, OR = 0.88, P = 0.428) (see Supplementary material online, 
Table S3). However, the publication rate of female senior scientists 
was markedly lower compared with the one of their male cowor- 
kers (18 vs. 31%, P = 0.004, Supplementary material online, Table 
S3). Concomitantly, female senior authorship was independently 
negatively associated with full-text publication in multivariable re- 
gression analysis (adjusted OR = 0.52, P = 0.010, Table 3). Scientif- 
ic impact, as assessed by 2-year citation rates, did not differ between 
genders (see Supplementary material online, Table S3). 

Discussion 

In this study, we identified independent predictors of acceptance at 
the ESC Congress, subsequent full-text publication and later 
impact through specific assessment and scientometric follow-up 
of a cohort of abstracts representative for over 10 000 abstracts 
submitted to an international scientific meeting (Figure 4). Pro- 
spective study design determined acceptance at Congress level, 
full-text publication, and subsequent impact after publication. Ran- 
domized controlled trials attracted almost seven times more 



2-year citations than other studies. Citation-based validation of 
the factors predicting acceptance at the Congress level or full-text 
publication reveal a potential discrimination of female senior 
authors at publication level and indicate superiority of blinded 
over open peer review. 

Identification of scientific quality at an 
early stage 

Presentation of abstracts at scientific meetings is an important part 
of the dissemination of knowledge. However, more than half of the 
studies submitted to scientific congresses are never published, 14 as 
is also the case for the ESC 2006. Identifying high-quality research 
based on distinct aspects when grading large numbers of abstracts 
is a challenging task for reviewers and editors. Reviewers and 
editors at the ESC Congress successfully identified science of sig- 
nificant later impact, given that 29.6% of all studies accepted for 
oral presentation received 10 or more 2-year citations compared 
with only 7.4% of rejected studies. Based on the assessment of 
the abstracts alone, we identified prospective study design, and 
specifically randomized controlled study design as strong predic- 
tors of scientific impact. 

Thus, key aspects of scientific quality are identifiable at the early 
stage of an abstract, and, even though markedly less pronounced, 
are taken into account in the review and editorial process of 
one of the largest international medical conferences. 

Hurdles to overcome from abstract to 
impact 

In the process of medical research, different stages are passed from 
the formulation of a well-substantiated hypothesis to a clearly 
proven and novel answer, and further to the acceptance of the ela- 
borated results in the scientific community. The first hurdle to 
overcome is publication. Intriguingly, the factors predicting 
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Table I Factors predicting acceptance at the ESC Congress 



Parameters 


No. of abstracts 


Acceptance 


Univariate reg. 


Multivatiate reg. 




submitted 


rate 


OR, 95% CI (P) 


OR, 95% CI (P) 




[% total (n)] 


[% total (n)] 






Overall rates 


100.0 (1002) 


37.5 (376) 






Type of research 










Clinical 


89.7 (899) 


36.8 (331) 


Ref. 


Ref. 


Basic 


10.3 (103) 


43.7 (45) 


1.33, 0.88-2.01 (0.174) 


2.24, 1.39-3.59 (0.001) 


1 \/r\f^ r^iT fnci"ii"i i1"i^n 

I Ul M liLI LU LIUI l 










Mrit 1 1 niwprci+v-afr i i ntp>H 

1 NUL U 1 1 1 V d jlly all ILIaLCU 


19.2 (192) 


34.9 (67) 


Ref. 




L J n i\/*^r*c il"\/«^Tr i i a1"£*H 
Ul II V CI il L¥ dl I I lldLCU 


80 8 T8101 


38 1 nfl91 

JO. 1 1 JU/I 


151 0 1 5-1 60 Id 4031 

l._) 1, \J. 1 J I.DU IUHUj 1 




Study design (clinical) 






(0.018) 


(0.023) 


I\CjLI UjUCLLIVC 


49.3 (494) 


32 4 C1601 


Ref. 


Ref. 


Pi~nc;nf i r1"i\/f 1 nnn.R ( I 

1 1 jLJC^_ LI VC 1 l"JI 1 1 \\ — . 1 


28 9 C2901 


42.8 (124) 


1 90 1 16-2 10 (0 016^ 

li/V) la IU Am * 1 VS 1 W ■ W 1 U 1 


1 67 1 22-2 29 (0 007\ 


RCT 


6.5 (65) 


47.7 (31) 


1.56, 1.13-2.21 (0.004) 


1.91, 1.11-3.28 (0.020) 


Meta-Analysis 


0.6 (6) 


50.0 (3) 


2.90, 0.42-10.46 (0.370) 


0.0 (1.000) 


Systematic Review 


0.2 (2) 


0.0 (0) 


0.0 (1.000) 


0.0 (1.000) 


Other 


4.2 (42) 


31.0 (13) 


n.a. 


n.a. 


Number of patients (clinical) 










<100 


43.4 (435) 


30.3 (132) 


Ref. 


Ref. 


>100 


41.7 (418) 


45.7 (191) 


1.93, 1.46-2.56 (<0.001) 


2.07, 1.54-2.77 (<0.001) 


Study field (clinical) 






(0.373) 




V dl Uk.IL 1 1 I kl.gl I lg, \-\J\ I IUU LdLIUI Idl, d.L.UlL. 


10 3 

1 W.J I 1 UJ 1 


44 7 (46) 


Ref. 




cardiac care 










Rhythmology 


12.7 (127) 


31.5 (40) 


0.57, 0.34-0.96 (0.034) 




I — <^>:irt fa i 1 1 i~p> loft \/F>n1"i"ii"i ilar fi inr1"ii~*n \/a \/i ilar 

1 ICdl L klllUI C IC 1 L VCI 1 LI LLUial 1 U 1 1 L LI <JI 1 , VaLVUlal 


15.2 (152) 


36 8 l'56'l 


071 044-1 16 (0 1731 

\J . / 1 | \J 1 1. IU 1 \J . 1 / J 1 




Hicpacp' niilmnnarv r'\m ila+inn 

LJ IjCojC) UUI! 1 1 KJ 1 Idl y L_ll LLJ Id LILJ 1 1 










Coronary artery disease, ischaemia 


13.2 (132) 


31.8 (42) 


0.61, 0.37-1.01 (0.055) 




Interventional cardiology, peripheral 


9.4 (94) 


39.4 (37) 


0.79, 0.47-1.36 (0.390) 




circulation, stroke 










Exercise, prevention, epidemiology, 


14.6 (146) 


36.3 (53) 


0.68, 0.41-1.12 (0.130) 




pharmacology, nursing 










Hypertension, myocardial and pericardial 


9.5 (95) 


38.9 (37) 


0.83, 0.48-1.42 (0.490) 




disease, cardiovascular surgery 










Other 


5.0 (50) 


40.0 (20) 


n.a. 


n.a. 


Gender first author b 










Male 


72.2 (723) 


39.8 (288) 


Ref. 


Ref. 


Female 


26.0 (261) 


33.7 (88) 


0.77, 0.57-1.03 (0.082) 


0.73, 0.54-1.00 (0.051) 


Gender last author c 










Male 


84.0 (842) 


38.2 (322) 


Ref. 




Female 


13.1 (131) 


38.9 (51) 


1.03, 0.71-1.50 (0.88) 





OR, odds ratio; CI, confidence interval; RCT, randomized controlled trial; ref, reference variable for odds ratio calculation; n.a., not applicable. Analyses performed on 
representative 10% selection (n — 1002), margin of error <0.01. Backward conditional variable exclusion, c-statistic — 0.620. 
Bold value indicates statistically significant findings. 

a ln 46 clinical studies, no information on the number of patients was provided. 
In 18 studies, no gender for the first author could be identified. 
c ln 29 studies, no gender for the last author could be identified. 



publication differed substantially from those predicting high citation 
rates. In essence, the institution where studies were conducted and 
the gender of the respective principal investigator predicted publi- 
cation — rather than the scientific content of a study. Such discrep- 
ancy hinders the flow of relevant findings to the scientific 



community and slows clinical advancement. Once published, the 
scientific community itself grades the scientific importance by 
either spreading the word through citation or not. This second 
and final step of appreciation by fellow scientists in the respective 
field was observed to depend only and substantially on research 
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Figure 3 Peer review and editorial process of the ESC Congress 2006. All abstracts submitted were peer-reviewed in a blinded fashion by 
three to eight expert reviewers, and graded on a scale from 1 to 10. (A) Reviewer ratings and editorial decisions. Accepted studies had received 
significantly higher ratings than rejected studies; studies for oral presentation had been ranked significantly higher than those for poster pre- 
sentations. The cut-off for acceptance at 6.5 was calculated using the Yuden's index (sensitivity 0.97, specificity 0.92). (8) Receiver operating 
characteristic (ROC) curve analyses after the scientometric follow-up of a representative subsample of 10% of all abstracts submitted 
(n = 1002) revealed a predictive value of average reviewer ratings of 69.4%. (Q Spearman's correlation of average reviewer ratings with accept- 
ance at the Congress, subsequent full-text-publication and citation rates uncovered a significant positive correlation in all three cases. (D) Com- 
parison of the numbers of 2-year citations of both accepted and rejected, subsequently published studies. Analyses revealed that rejected and 
subsequently published studies were cited significantly less frequently compared with accepted and subsequently published studies; citation 
frequencies between different presentation formats did not differ. AUC, area under the curve, significance level (P < 0.017) adjusted for mul- 
tiple comparisons using Bonferroni's correction, interquartile ranges, whiskers indicate minima and maxima. 



design. Considering that in current evidence-based medicine clinic- 
al guidelines are set forth based on the respective level of 
evidence, 15 which is primarily based on study design, 16 this result 
is encouraging. Moreover, our findings indicate that judging scien- 
tific quality of medical research based on other factors, such as 
the number of patients enrolled, the home institution, or the 
gender of the performing researchers, may not be justified. 

Peer review and editorial decisions 

Expert peer review is the gold standard for research evaluation 
when it comes to publication in peer review journals or allocation 



of resources. 17 Given its importance, surprisingly little research has 
been performed in this field. 18,19 Abstracts submitted to the ESC 
Congress 2006 were subjected to ('double'-) blinded peer 
review (neither authors knew reviewers nor vice versa). Up to 
eight independent reviewers graded scientific quality on a scale 
from 1 to 10, providing the basis for editorial decision. Information 
on sound criteria to specifically evaluate when grading scientific 
quality is scarce. 20 In this study, we assessed putative determinants 
of scientific quality, and identified four aspects that were in the 
focus of the peer review and editorial process of the ESC Congress 
2006. External validation through citation analyses confirmed two 
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Table 2 Determination of the best cut-off of the average reviewer rating for frequent citation 







Frequency 


Accepted at 


Published 


> 1 0 two year 


P-value 


OR (95% CI) 


AUC 






III 1 /D3UUII IIIICU II 


rnnorocc V rt ( ®/L 

LUIIglCM III I/O 
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III I/O WILIIIII 


rit Tn (% within 

I.IL> III I/O WILIIIII 


(Fisher's 




(ROC 








within rating)] 


rating)] 


rating)] 


exact) 




analyses) 


Av. Rev. Rating 


> 6 


555 (55.4) 


375 (67.6) 


1 QQ /"DC Q\ 

199 (35.9) 


36 (6.5) 


0.036 


2.34/ (1.U4— 5.2/) 


0.584 


Av. Rev. Rating 


> 6.5 


394 (39.3) 


358 (90.9) 


146 (37.1) 


32 (8.1) 


0.002 


3.135 (1.53-6.37) 


0.634 


Av. Rev. Rating 


>7 


213 (21.3) 


210 (98.6) 


81 (38.0) 


24 (11.3) 


<0.001 


4.021 (2.07-7.80) 


0.660 


Av. Rev. Rating 


>7.5 


111 (11.1) 


110 (99.1) 


47 (42.3) 


15 (13.5) 


0.001 


3.491 (1.69-7.21) 


0.609 


Av. Rev. Rating 


>8 


33 (3.3) 


32 (97.0) 


15 (45.5) 


6 (18.2) 


0.015 


4.193 (1.41-12.44) 


0.553 


Av. Rev. Rating 


> 8.5 


12 (1.2) 


11 (91.7) 


5 (41.7) 


4 (33.3) 


0.002 


24.700 (2.69-226.65) 


0.543 


Av. Rev. Rating 


>9 


Not given 















P-values, odds ratios (OR), confidence intervals (CI) and area under the curve (AUC) values refer to the categorigal variable of '> 10 two-year citations'. Random 1 0% selection 
(n = 1002), margin of error <0.01. 

Bold value indicates statistically most significant finding in conjunction with the largest AUC. 



out of these four aspects to be markers of scientific impact, indi- 
cating effective peer review and editorial work at this stage. More- 
over, a clear cut-off for average reviewer ratings with regard to 
acceptance or rejection was identified. Along with significantly 
higher ratings of abstracts accepted for oral compared with 
those accepted for poster presentation, these data underscore a 
coherent cooperation of reviewers and editors. Scientometric 
follow-up assigned peer reviewer ratings a predictive value of 
70% for later scientific impact, further supporting effective peer 
review and editorial work at Congress level. 

Assessment of the predictive value of the same set of para- 
meters for subsequent full-text publication revealed markedly dif- 
ferent factors. External validation through citation analyses did not 
confirm any of the identified predicting factors to be markers of 
scientific impact: surprisingly, randomized controlled trials design 
did not predict publication. Moreover, gender and institutional af- 
filiation of the senior author were related to whether or not an ab- 
stract led to a full-text publication. As neither gender nor 
institutional affiliation delineate scientific quality, a potential bias 
during single-blinded review cannot be excluded. 

Blinded peer review and scientific quality 

Most scientific and medical journals use 'anonymous' or 'single- 
blinded' peer review, i.e. the authors do not know the reviewers. 
However, only a minority applies 'blinded' or 'double-blinded' peer 
review, where neither authors know reviewers nor vice versa. 3 Dif- 
ferent kinds of bias are recognized as confounding factors in the 
effort to judge the quality of research. 4 ' 17 ' 18,21,22 When the American 
Heart Association adopted blinded peer review for their Scientific 
Sessions, Ross et atV demonstrated that blinded peer review at 
least partially reduced reviewer bias with regard to prestigious insti- 
tutions or author origin. However, it remained unclear whether the 
reduction in bias by adopting blinded peer review would affect later 
scientific impact. In the present study, separate validation of blinded 
peer review at Congress level and presumably largely single-blinded 
peer review at publication level suggests substantial enhancement of 
scientific quality by decreasing potential bias through blinded peer 
review. Given these data and other reports that blinded peer 



review does not only reduce bias but improve scientific quality, it 
can be speculated that blinded peer review merits consideration 
on a broader scale. 

Cardiovascular science and gender 

The number of women in medical training and entry-level academ- 
ic positions has substantially increased in recent years, whereas 
their representation in advanced and leading academic positions 
has remained comparatively low. 24-27 One of the empirically 
best proven hypotheses for this disparity is that women would 
publish fewer papers and consequently raise less soft money 
compared with their male co-workers. 26 Among all abstracts 
submitted to the ESC Congress 2006, 26% originated from 
female first authors, and only 13% from senior female authors. 
Hence, women appear to be underrepresented in both the entry 
level and advanced academic positions in cardiovascular medicine. 

There were no gender differences with regard to abstract ac- 
ceptance in blinded peer review, underlining the absence of qualita- 
tive differences, as supported by others. 25 Interestingly, the rate of 
full-text publication of female senior authors was significantly lower 
compared with their male colleagues (19.1 vs. 30.8%). Concomi- 
tantly, senior female authorship was identified as independent nega- 
tive predictor of full-text publication in multivariable regression 
analysis. The scientometric follow-up did not reveal any gender 
effects, again supporting the absence of qualitative disparities. 
Since we did not assess the number of submissions, but only of suc- 
cessful full-text publications, we cannot exclude that female senior 
authors have submitted their research less frequently for full-text 
publication. Female underrepresentation, and the fact that citation 
analyses in this study aimed to assess whether or not a published 
article was among the top 1 5% of published articles, limit the inter- 
pretation of female scientific performance. However, the discrep- 
ancy between blinded peer review at the Congress level and 
largely open peer review at the publication level in this regard 
may alert the scientific community to shed light on a potential bias. 

Of course, all of the above findings should be interpreted cau- 
tiously and considered exploratory, given that information on the 
gender of first and last authors was not directly available but had 
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Table 3 Factors predicting publication 



Parameters 
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No. of studies 


Publication Rate 


Univ. reg. OR, 


Multiv. reg. OR, 
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not published 


(% submitted 3 ) 


95% CI (P) 


95% CI (P) 




published 










Type of research 
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Ref. 
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RCT 


22 


43 


33.8 


1.53, 0.88-2.65 (0.133) 


1.48, 0.84-2.61 (0.181) 


Meta-Analysis 


4 


2 


66.7 


5.97, 1.08-32.98 (0.041) 


0.0 (1.000) 


Systematic Review 


0 


2 


0.0 


0.0 (1.000) 


0.0 (1.000) 


Other 


11 


31 


26.2 


n.a. 


n.a. 


Number of patients (clin.) 












<100 


120 


318 


27.4 


Ref. 




>100 


117 


298 


28.2 


1.08, 0.80-1.46 (0.068) 




btudy field (clinical) 








\U.AAO j 




Cardiac imaging, computational, 


32 


71 


31.1 


Ref. 




acute cardiac care 












Rhythmology 


30 


97 


23.6 


0.71, 0.41-1.25 (0.236) 




Heart failure, left ventricular 


40 


112 


26.3 


0.86, 0.51-1.47 (0.588) 




function, valvular disease, 












pulmonary circulation 












Coronary artery disease, 


39 


93 


29.5 


0.94, 0.55-1.61 (0.821) 




ischaemia 












Interventional cardiology, 


27 


67 


28.7 


0.86, 0.47-1.55 (0.610) 




peripheral circulation, stroke 












Exercise, prevention, 


36 


110 


24.7 


0.76, 0.44-1.32 (0.328) 




epidemiology, pharmacology, 












nursing 












Hypertension, myocardial and 
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pericardial disease, 












cardiovascular surgery 












Other 


18 


32 


36.0 


n.a. 


n.a. 


Gender first author c 












Male 


217 


506 


30.0 


Ref. 


Ref. 


Female 


71 


190 


27.2 


0.88, 0.63-1.18 (0.352) 


0.94, 0.67-1.32 (0.723) 


Gender last author d 












Male 


259 


583 


30.8 


Ref. 


Ref. 


Female 


25 


106 


19.1 


0.54, 0.34-0.85 (0.008) 


0.52, 0.32-0.85 (0.010) 



OR, odds ratio; CI, confidence interval; RCT, randomized controlled trial; ref, reference variable for odds ratio calculation; n.a., not applicable. Analyses performed on 

representative 10% selection (n — 1002), margin of error < 0.01. Backward conditional variable exclusion, c-statistic — 0.757. 

Bold value indicates statistically significant findings. 

a Percent submitted refers to submissions to the ESC Congress. 

b ln 46 clinical studies, no information on the number of patients was provided. 

c ln 18 studies, no gender for the first author could be identified. 

d ln 29 studies, no gender for the last author could be identified. 
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Table 4 Factors predicting high citation rates 



Parameters 


Articles cited 10 x or 


Univariate reg. OR, 


Multivatiate reg. OR, 




more in 2 years 


95% CI (P) 


95% CI (P) 




[% of published (n/n)] 






Type of research 
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Coronary artery disease, ischemia 


30.8 (12/39) 


1.96, 0.66-5.84 (0.229) 


.79, 0.54-5.37 (0.331) 


Interventional cardiology, peripheral circulation, 


25.9 (7/27) 


1.86, 0.56-2.49 (0.310) 


.44, 0.44-5.72 (0.582) 


stroke 








Exercise, prevention, epidemiology, 


56 (2/36) 


0.69, 0.19-2.49 (0.567) 


0.71, 0.19-2.95 (0.630) 


pharmacology, nursing 








Hypertension, myocardial and peri-cardial 


6.9 (2/29) 


0.42, 0.10-1.62 (0.254) 


0.39, 0.09-1.90 (0.230) 


disease, cardiovascular surgery 








Other 


5.6 (1/18) 


n.a. 


n.a. 


Gender first author b 








Male 


14.3 (31/217) 


Ref. 


Ref. 


Female 


18.3 (13/71) 


1.34, 0.66-2.73 (0.423) 


0.0 (1.00) 


Gender last author c 








Male 


16.6 (43/259) 


Ref. 




Female 


4.0 (1/25) 


0.22, 0.03-1.66 (0.142) 





OR, odds ratio; CI, confidence interval; RCT, randomized controlled trial; ref., reference variable for odds ratio calculation, n.a., not applicable. Analyses performed on 
representative 10% selection (n — 1002), margin of error <0.01. Backward conditional variable exclusion, c-statistic — 0.570. 
Bold value indicates statistically significant findings. 

a ln 46 clinical studies, no information on the number of patients was provided. 
b ln 18 studies, no gender for the first author could be identified. 
c ln 29 studies, no gender for the last author could be identified. 



to be determined using a systematic approach. Given that this 
approach was applied a priori (i.e. before any of the analyses 
were conducted), however, we believe it is unlikely to lead to 
any systematic biases and, if anything, misclassification would 
result in findings shifted toward no differences. That said, the im- 
portance of understanding the role of gender in research is 



critical and certainly requires further confirmation with additional 
work. 

Limitations 

In all regression analyses performed in this study, potential con- 
founding factors that were not assessed or included in the 



3044 



S. Winnik et ai 



prediction 


validation 




Acceptance 




igress 


1 . basic research 

(OR 2.2. 95% C.I. 1.4-3.5) 




2. patient number > 100 

(OR 2.1, 95% C.I. 1.5-2.8) 




o 
o 


3. prospective non-randomized study design 

(OR 1.7. 95% C.I. 1.2-2.3) 


JL 

T 




4. randomized controlled trial design 

(OR 1.9. 95% C.I. 1.1 -3.3) 


+ 

■ 


c 
o 


Full-text publication 

1 . basic research 

(OR 2.1. 95% C.I. 1.3-3.3) 


• 


jblical 


2. institutional affiliation 

(OR 1.6. 95% C.I. 1.1 -2.4) 


% 


Q. 


3. female senior authorship 
negative association 

(OR 0.5. 95% C.I. 0.3 - 0.9) 


% 




Impact 




citation 


1. randomized controlled trial design 

(OR 6.8, 95% C.I. 2.5-21.1) 

2. prospective non-randomized study design 

(OR 2.6, 95% C.I. 1.2-5.7) 





Figure 4 Key findings: predictors of acceptance at Congress level, full-text publication, and frequent citation, and their validation with regard 
to scientific impact. We identified parameters predicting acceptance, subsequent full-text publication, and high citation rates (>10 two-year 
citations). Parameters predicting scientific impact (+), no association with scientific impact ( — ). All findings, odds ratios, and confidence inter- 
vals (CI) refer to multivariable regression analyses based on a random 10% selection of all abstracts submitted to the ESC Congress 2006 
(n = 1002), margin of error < 0.01. 



respective regression analysis cannot be excluded. Therefore, 
extrapolations have to be done with care. For example, although 
no predictors of oral presentation among accepted studies were 
identified, other aspects such as novelty, which was not assessed 
in this study, may also play a role. Of note, the presentation 
format at the Congress level is also subject to organizational 
aspects, including the arrangement of consistent themes for scien- 
tific sessions. 

Even though citation analysis is a widely accepted and neutral 
tool in evaluative bibliometrics to provide quantitative indicators 
for scientific performance, 17,28-31 we are aware that it may be 
subject to bias. An article published in a very renowned journal 
or authored by a well-known expert may enhance the number 
of its citations compared with a similar article published in a less 
renowned journal or authored by a less known scientist. Citation 
analysis can, of course, not replace but only complement expert 
peer review in the evaluation of research. However, in the 
context of this study, we consider citation suitable as an external 
quality assessment in the evaluation of a large number of abstracts. 
Due to the extensive amount of studies, all authored by multiple 



investigators, self-citation was not assessed. Importantly, no signifi- 
cant association between institutional affiliation and the number of 
citations of a published article was observed, indicating that this 
shortcoming is of minor importance for the respective results. 

The chosen publication and citation windows, i.e. 2 years each, 
may not cover all full-text publications among the studies assessed. 
For the rapid field of biomedical research, however, a publication 
and citation window of 2 years each has been described to 
cover the majority of publications and citation peaks. 9,17,30 ' 31 Fur- 
thermore, the assessment of the peer review and editorial 
process at publication level is based on the assumption that 
studies that were not published in the 4 years following the Con- 
gress had been submitted for full-text publication and been 
rejected — which was not specifically assessed in this study. In 
essence, authors of those studies identified as not published 
were not contacted to verify that the respective studies had 
been submitted for publication in a peer review journal. Therefore, 
the results of the logistic regression analysis on the level of publi- 
cation may be affected by factors that led to non-submission as 
opposed to reviewer ratings and editorial decisions. Consequently, 
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extrapolations regarding peer review and editorial process at pub- 
lication level have to be interpreted with care. 

The discussed potential effect of anonymized peer review at 
publication level is based on the work of others, stating that less 
than 20% of almost all English language scientific and medical jour- 
nals use double-blinded peer review. 3 Editorial offices of all re- 
spective journals in this study sample have not been contacted. 
As noted above, author gender was assessed using an algorithm 
based on first- and surnames in addition to photographs where ne- 
cessary and available. The related findings may consequently be 
subject to bias based on the indirect methods used. 

Generalization of the data of this study, raised in the cardiovas- 
cular field, to other medical specialties has to be handled with 
caution. Nevertheless, given the diversity and quantity of cardio- 
vascular science worldwide as well as the size of the cohort in 
this study, our findings may be of interest to a broader audience 
of scientists in different fields of medical research. 

Conclusions and possible room for 
improvement 

High quality-research was determined by prospective study design 
that was readily identified at Congress level. Average reviewer 
ratings predicted frequent citation rates, indicating an effective 
evaluation process at Congress level. The scientometric follow-up 
revealed an unexpected disparity between predictors of publica- 
tion and citation, suggesting a disadvantage of non-academic insti- 
tutions and female senior authors at publication level. The 
observed discrepancy between blinded peer review at Congress 
level and largely single-blinded peer review at publication level indi- 
cates that blinding handling editors and reviewers at publication 
level may merit a closer look. 

Supplementary material 

Supplementary material is available at European Heart Journal online. 

Acknowledgements 

We are grateful to Florence Biazeix for administrative assistance, 
and to Eirini Liova for data collection and management. 

Funding 

This work was funded by the Foundation for Cardiovascular Research, 
Zurich, Switzerland, the Swiss National Science Foundation 
310030-130626/1 (C.M.M.), and the University Research Priority 
Program' Integrative Human Physiology' at the University of Zurich 
(P.-A.C, T.F.L, C.M.M.). The views expressed in this publication are 
those of the authors and not necessarily of the funding organizations. 
Funding to pay the Open Access publication charges for this article was 
provided by the Foundation for Cardiovascular Research (FCR) in 
Zurich, Switzerland. 

Conflict of interest: none declared. 



References 

1. The European Society of Cardiology E. ESC Congresses Reports. http://www 
.escardio.org/congresses/esc-2011/Pages/welcome.aspx. 

2. The European Society of Cardiology E. Past ESC Congresses. http://www.escardio 
.org/congresses/past_congresses/Pages/past-ESC-congresses.aspx. 

3. Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of re- 
search papers by peer review. JAMA 1994;272:143-146. 

4. GarfunkelJM, Ulshen MH, HamrickHJ, Lawson EE. Effect of institutional prestige on 
reviewers' recommendations and editorial decisions. JAMA 1994;272:137-138. 

5. Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, 
Reiling J, Pace B. Publication bias in editorial decision making. JAMA 2002;287: 
2825-2828. 

6. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias 
and other limitations in the outcome of research abstracts submitted to a scien- 
tific meeting. JAMA 1998;280:254-257. 

7. Krzyzanowska MK, Pintilie M, Tannock IF. Factors associated with failure to 
publish large randomized trials presented at an oncology meeting. JAMA 2003; 
290:495-501. 

8. Raptis DA, Oberkofler CE, Gouma D, Garden OJ, Bismuth H, Lerut T, 
Clavien PA. Fate of the peer review process at the ESA: long-term outcome of 
submitted studies over a 5-year period. Ann Surg 2010;252:715-725. 

9. Scherer RW, Dickersin K, Langenberg P. Full publication of results initially pre- 
sented in abstracts. A meta-analysis. JAMA 1994;272:158-162. 

10. Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, Armstrong PW. Tran- 
sition from meeting abstract to full-length journal article for randomized con- 
trolled trials. JAMA 2006;295:1281-1287. 

11. Ross JS, Gross CP, Desai MM, Hong Y, Grant AO, Daniels SR, Hachinski VC, 
Gibbons RJ, Gardner TJ, Krumholz HM. Effect of blinded peer review on abstract 
acceptance. JAMA 2006;295:1 675-1 680. 

12. Bartlett JE, Kotrlik JW, Higgins CC. Organizational research: determining appro- 
priate sample size in survey research. Info Techno! Learn Perform J 2001;19:43-50. 

1 3. Reuters T. ISI Web of Knowledge. http://apps.webofknowledge.com/UA_ 
GeneralSearch_input.do?product=UA&search_mode=GeneralSearch&S!D = 
4B7Nb@C@ED49jmlB51P&preferencesSaved=. 

14. von Elm E, Costanza MC, Walder B, Tramer MR. More insight into the fate of bio- 
medical meeting abstracts: a systematic review. BMC Med Res Methodol 2003;3:12. 

15. Glasziou P, Vandenbroucke JP, Chalmers I, Assessing the quality of research. BMJ 
2004;328:39-41. 

16. Medicine TOCfE-B. Levels of Evidence-based Medicine, http://www.cebm.net/index 
.aspx?o=1025 (last update 2009; date last accessed). 

17. Pendlebury DA. The use and misuse of journal metrics and other citation indica- 
tors. Arch Immunol Ther Exp (Warsz) 2009;57:1-11. 

18. Kassirer JP, Campion EW. Peer review. Crude and understudied, but indispens- 
able. JAMA 1994;272:96-97. 

19. Bailar JC 3rd, Patterson K. The need for a research agenda. N Engl J Med 1985; 
312:654-657. 

20. Abby M, Massey MD, Galandiuk S, Polk HC Jr. Peer review is an effective screen- 
ing process to evaluate medical manuscripts. JAMA 1994;272:105-107. 

21. Relman AS, Angell M. How good is peer review? N Engl J Med 1989;321:827-829. 

22. Link AM. US and Non-US submissions: an analysis of reviewer bias. JAMA 1994; 
272:246-247. 

23. Laband DN, Piette Mj. A citation analysis of the impact of blinded peer review. 
JAMA 1994;272:147-149. 

24. Ash AS, Carr PL, Goldstein R, Friedman RH. Compensation and advancement of 
women in academic medicine: is there equity? Ann Intern Med 2004;141: 
205-212. 

25. Tesch BJ, Wood HM, Helwig AL, Nattinger AB. Promotion of women physicians 
in academic medicine. Glass ceiling or sticky floor? JAMA 1995;273:1022-1025. 

26. Kaplan SH, Sullivan LM, Dukes KA, Phillips CF, Kelch RP, Schaller JG. Sex differ- 
ences in academic advancement. Results of a national study of pediatricians. N 
Engi J Med 1996;335:1282-1289. 

27. Levinson W, Lurie N. When most doctors are women: what lies ahead? Ann Intern 
Med 2004;141:471-474. 

28. Garfield E. Citation indexes for science; a new dimension in documentation 
through association of ideas. Science 1955;122:108-111. 

29. Garfield E. Is citation a legitimate evaluation tool? Sdentometrics 1979;1:359-375. 

30. Haeffner-Cavaillon N, Graillot-Gak C. The use of bibliometric indicators to help 
peer-review assessment. Arch Immunol Ther Exp (Warsz) 2009;57:33-38. 

31. Moed HF. New developments in the use of citation analysis in research evalu- 
ation. Arch Immunol Ther Exp (Warsz) 2009;57:13-18. 



